Information Retrieval Using Statistical Classification

In the classical information retrieval IR problem the system must nd all docu ments in a collection that are related to a topic de ned by a user s query A common approach to the IR problem is to represent documents and the query as vectors of term frequencies and rank the documents in the collection according to their inner product similarity with respect to the query When a sample of evaluated docu ments is available in addition to the query often called routing the problem can be attacked using techniques based on statistical classi cation In order for statistical classi cation to be a feasible approach the system must produce a relatively small set of high quality feature variables It turns out that individual words due to their quantity and ambiguity are not optimal features Previous work has focused on a technique known as Latent Semantic Indexing LSI which applies the singular value decomposition to a term document matrix and represents terms and documents by linear combinations of orthogonal indexing variables The research presented in this thesis accomplishes the following goals It provides a thorough discussion of evaluation in information retrieval experiments It intro duces the concept of a local LSI decomposition LSI is used separately on a set of documents in the local region surrounding each query creating query speci c feature variables and making the LSI technique feasible for very large document collections It applies the classi cation technique known as Discriminant Analysis to the routing problem and presents experimental results on two text collections It demonstrates that using a local LSI decomposition improves retrieval performance and represents documents using a relatively small number of feature variables It nds that Dis criminant Analysis sometimes leads to additional performance gains but that more research is needed to determine the optimal size and shape of the local region